Emerging studies have emphasized the importance of the fidelity of cortical representation in forming enduring episodic memory. No study, however, has examined whether there are age-related reductions in representation fidelity that can explain memory declines in normal aging. Using functional MRI and multivariate pattern analysis, we found that older adults showed reduced representation fidelity in the visual cortex, which accounted for their decreased memory performance even after controlling for the contribution of reduced activation level. This reduced fidelity was specifically due to older adults’ poorer item-specific representation, not due to their lower activation level and variance, greater variability in neuro-vascular coupling, or decreased selectivity of categorical representation (i.e., dedifferentiation). Older adults also showed an enhanced subsequent memory effect in the prefrontal cortex based on activation level, and their prefrontal activation was associated with greater fidelity of representation in the visual cortex and better memory performance. The fidelity of cortical representation thus may serve as a promising neural index for better mechanistic understanding of the memory declines and its compensation in normal aging.